CLDSRV-968: Expire idle keep-alive sockets in the functional test client by tcarmet · Pull Request #6247 · scality/cloudserver

tcarmet · 2026-08-04T22:37:02Z

The functional test client keeps pooled keep-alive connections indefinitely, so after an idle gap it can send a request over a connection the server has already closed and fail with a spurious connection reset. Because the client also runs with retries disabled, one reset is enough to fail a hook and take a whole group of tests down with it, which surfaces as intermittent CI red on slower full-deployment environments where such idle gaps actually occur.

This makes the client expire its own idle connections before the server does, so the reuse race cannot arise.

bert-e · 2026-08-04T22:37:06Z

Hello tcarmet,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
`/after_pull_request`	Wait for the given pull request id to be merged before continuing with the current one.
`/bypass_author_approval`	Bypass the pull request author's approval	⭐
`/bypass_build_status`	Bypass the build and test status	⭐
`/bypass_commit_size`	Bypass the check on the size of the changeset `TBA`	⭐
`/bypass_incompatible_branch`	Bypass the check on the source branch prefix	⭐
`/bypass_jira_check`	Bypass the Jira issue check	⭐
`/bypass_peer_approval`	Bypass the pull request peers' approval	⭐
`/bypass_leader_approval`	Bypass the pull request leaders' approval	⭐
`/approve`	Instruct Bert-E that the author has approved the pull request.		✍️
`/create_pull_requests`	Allow the creation of integration pull requests.
`/create_integration_branches`	Allow the creation of integration branches.
`/no_octopus`	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
`/unanimity`	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
`/wait`	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
`/help`	Print Bert-E's manual in the pull request.
`/status`	Print Bert-E's current status in the pull request `TBA`
`/clear`	Remove all comments from Bert-E from the history `TBA`
`/retry`	Re-start a fresh build `TBA`
`/build`	Re-start a fresh build `TBA`
`/force_reset`	Delete integration branches & pull requests, and restart merge process from the beginning.
`/reset`	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

bert-e · 2026-08-04T22:37:12Z

Incorrect fix version

The Fix Version/s in issue CLDSRV-968 contains:

None

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

9.3.16
9.4.0

Please check the Fix Version/s of CLDSRV-968, or the target
branch of this pull request.

bert-e · 2026-08-05T15:11:24Z

Request integration branches

Waiting for integration branch creation to be requested by the user.

To request integration branches, please comment on this pull request with the following command:

/create_integration_branches

Alternatively, the /approve and /create_pull_requests commands will automatically
create the integration branches.

tcarmet · 2026-08-05T15:45:58Z

/create_integration_branches

bert-e · 2026-08-05T15:46:10Z

Integration data created

I have created the integration data for the additional destination branches.

this pull request will merge bugfix/CLDSRV-968-test-client-idle-socket-timeout into
development/9.3
w/9.4/bugfix/CLDSRV-968-test-client-idle-socket-timeout will be merged into development/9.4

The following branches will NOT be impacted:

development/7.10
development/7.4
development/7.70
development/8.8
development/9.0
development/9.1
development/9.2

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

The following options are set: create_integration_branches

bert-e · 2026-08-05T15:46:11Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

The following options are set: create_integration_branches

bert-e · 2026-08-06T17:44:15Z

Incorrect fix version

The Fix Version/s in issue CLDSRV-968 contains:

9.3.16
9.4.0

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

9.3.17
9.4.0

Please check the Fix Version/s of CLDSRV-968, or the target
branch of this pull request.

The following options are set: create_integration_branches

The SDK agents in the aws-node-sdk test client enable keepAlive without an idle timeout, so pooled sockets are never expired client-side. Node's HTTP server closes idle keep-alive connections after 5s and advertises `Keep-Alive: timeout=5`, but the agent only honours that hint when it already has a non-zero timeout of its own. Without one, a burst of requests issued after an idle period can be written to connections the server has already closed, failing with ECONNRESET ("socket hang up"). Set a 4s agent idle timeout so the client retires sockets before the server does. This only destroys sockets sitting in the free pool; in-flight requests stay governed by requestTimeout.

tcarmet · 2026-08-06T17:46:33Z

/reset

bert-e · 2026-08-06T17:46:42Z

Reset complete

I have successfully deleted this pull request's integration branches.

The following options are set: create_integration_branches

bert-e · 2026-08-06T17:47:42Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

The following options are set: create_integration_branches

tcarmet · 2026-08-06T17:48:22Z

/approve

bert-e · 2026-08-06T18:13:18Z

Build failed

The build for commit did not succeed in branch w/9.4/bugfix/CLDSRV-968-test-client-idle-socket-timeout

The following options are set: approve, create_integration_branches

bert-e · 2026-08-06T18:21:49Z

Build failed

The build for commit did not succeed in branch bugfix/CLDSRV-968-test-client-idle-socket-timeout

The following options are set: approve, create_integration_branches

codecov · 2026-08-06T20:00:28Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.08%. Comparing base (81c3ce1) to head (7346130).
⚠️ Report is 15 commits behind head on development/9.3.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@               Coverage Diff                @@
##           development/9.3    #6247   +/-   ##
================================================
  Coverage            85.08%   85.08%           
================================================
  Files                  206      206           
  Lines                13422    13422           
================================================
  Hits                 11420    11420           
  Misses                2002     2002

Flag	Coverage Δ
file-ft-tests	`68.38% <ø> (+0.05%)`	⬆️
kmip-ft-tests	`28.37% <ø> (ø)`
mongo-v1-ft-tests	`69.57% <ø> (ø)`
s3c-ft-tests-v0	`64.08% <ø> (ø)`
s3c-ft-tests-v1	`64.05% <ø> (ø)`
unit	`71.00% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

bert-e · 2026-08-06T20:00:40Z

Build failed

The build for commit did not succeed in branch w/9.4/bugfix/CLDSRV-968-test-client-idle-socket-timeout

The following options are set: approve, create_integration_branches

bert-e · 2026-08-06T20:34:26Z

Build failed

The build for commit did not succeed in branch bugfix/CLDSRV-968-test-client-idle-socket-timeout

The following options are set: approve, create_integration_branches

tcarmet marked this pull request as ready for review August 4, 2026 23:08

tcarmet requested a review from a team August 4, 2026 23:08

leif-scality approved these changes Aug 5, 2026

View reviewed changes

tcarmet requested a review from a team August 5, 2026 15:45

anurag4DSB approved these changes Aug 5, 2026

View reviewed changes

jonathan-gramain reviewed Aug 5, 2026

View reviewed changes

Comment thread tests/functional/aws-node-sdk/test/support/config.js

BourgoisMickael approved these changes Aug 5, 2026

View reviewed changes

Comment thread tests/functional/aws-node-sdk/test/support/config.js Outdated

tcarmet force-pushed the bugfix/CLDSRV-968-test-client-idle-socket-timeout branch from 0065a20 to 7346130 Compare August 6, 2026 17:46

scality deleted a comment from bert-e Aug 6, 2026

Uh oh!

Conversation

tcarmet commented Aug 4, 2026

Uh oh!

bert-e commented Aug 4, 2026

Hello tcarmet,

Uh oh!

bert-e commented Aug 4, 2026

Incorrect fix version

Uh oh!

bert-e commented Aug 5, 2026

Request integration branches

Uh oh!

tcarmet commented Aug 5, 2026

Uh oh!

bert-e commented Aug 5, 2026

Integration data created

Uh oh!

bert-e commented Aug 5, 2026

Waiting for approval

Uh oh!

Uh oh!

Uh oh!

bert-e commented Aug 6, 2026

Incorrect fix version

Uh oh!

tcarmet commented Aug 6, 2026

Uh oh!

bert-e commented Aug 6, 2026

Reset complete

Uh oh!

bert-e commented Aug 6, 2026

Waiting for approval

Uh oh!

tcarmet commented Aug 6, 2026

Uh oh!

bert-e commented Aug 6, 2026

Build failed

Uh oh!

bert-e commented Aug 6, 2026

Build failed

Uh oh!

codecov Bot commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

bert-e commented Aug 6, 2026

Build failed

Uh oh!

bert-e commented Aug 6, 2026

Build failed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

codecov Bot commented Aug 6, 2026 •

edited

Loading